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DETAILED ACTION 
Response to Amendment 

1 . In response to the Office Action mailed August 7, 2007, applicant submitted an 
amendment filed on November 1 1 , 2007, in which the applicant amended and 
requested reconsideration. 

Response to Arguments 

2. Applicant argues that the prior art cited does not teach wherein said audio 
annotation is in the form of audio clips and wherein the audio annotation is 
associated with an author of said audio annotation , said audio clips are stored in a 
storage which includes properties that permit audio information to be associated 
with a visual so that one audio clip is displayed in synchronism with another audio 
clip and an audio input for receiving verbally delimited keywords and converting said 
verbally delimited keywords into search queries and wherein the user interface is 
controlled by a processor which allows dynamically accessing the audio clips based 
on the search gueries . 

Firstly, Applicant argues that Schilit is distinguished from the claimed invention in 
that none of the storage devices include any properties that permit audio information to 
be associated with a visual so that one audio clip is displayed in synchronism with 
another audio clip. However, nowhere in the Applicant's specification does it teach one 
audio clip being displayed in synchronism with another audio clip. According the 
specification (PGPUB 2002/0099552), paragraphs 0045, 0066, 0104 and 0141, it 
teaches synchronizing video with audio, but not audio with audio. 
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Further, Applicant argues that Schilit is completely silent on whether the 
annotations are in the form of audio clips . However, Schilit was not used to teach 
annotation in the form of audio clips. Regardless, the limitation has rejected in view of 
new grounds of rejections. 

Applicant also argues that Schilit does not receive verbally delimited keywords 
and converting said verbally delimited keywords into search queries. Applicant claims 
verbally delimiting keywords, however, it is unclear whether the keywords are being 
verbally delimited or if the verbal keywords are delimited. According to the specification 
paragraph 0098, it teaches that delimiters are added to the audio clip or annotation 
when special emphasis is used of a word or words, not verbally delimited. 

Claim Rejections - 35 USC §112 

3. The following is a quotation of the first paragraph of 35 U.S.C. 112: 

The specification shall contain a written description of the invention, and of the manner and process of 
making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the 
art to which it pertains, or with which it is most nearly connected, to make and use the same and shall 
set forth the best mode contemplated by the inventor of carrying out his invention. 

4. Claims 1,10, 27, 33, 43, 50, 55, 70, 73, 75, 77, 83 and 85 are rejected under 35 
U.S.C. 112, first paragraph, as failing to comply with the written description requirement. 
The claim(s) contains subject matter which was not described in the specification in 
such a way as to reasonably convey to one skilled in the relevant art that the 
inventor(s), at the time the application was filed, had possession of the claimed 
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invention. In particular, synchronizing one audio clip with another audio clip and 
verbally delimiting keywords. 

Claim Rejections - 35 USC § 103 

5. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

6. Claims 27-28 and 31-35, 39-46, 48, 50-56, 58, 60, 62-63, 66-73, 75 and 77, 81- 

86 are rejected under 35 U.S.C. 103(a) as being unpatentable over Schilit et al. (USPN 
6,279,014), hereinafter referenced as Schilit in view of Mishra et al. (USPN 5,805,118), 
hereinafter referenced as Mishra in further view of Oliver (USPN 6,226,422) and in 
further view of Lucas et al. (USPN 6,012,074), hereinafter referenced as Lucas. 

Regarding claim 27, Schilit discloses a user interface for displaying electronic 
information to a user comprising: 

a first display portion for displaying a portion of a document (pictures themselves; 
column 6, lines 17-19); and 

a second display portion for displaying a graphical indication that said document 
includes an audio annotation associated with said displayed portion of said document 
(annotations of pictures; column 6, lines 17-19), but does not specifically teach wherein 
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said audio annotation is associated with an author, wherein the audio annotations are in 
the form of audio clips, verbally delimiting keywords and dynamically accessing the 
audio clips based on search queries. 

Mishra teaches a user interface for displaying electronic information to a user 
wherein said audio annotation is associated with an author (column 12, line 59 - column 
13, line 45 with column 19, lines 8-17 and column 28, line 61 - column 29, line 22), so 
that the marks are distinguishable. 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to modify Schilit's interface wherein said audio annotation 
is associated with an author, as taught by Mishra, to make it helpful to others to 
distinguish marks made by user A from those of user B and even a user to distinguish 
his or her down earlier remarks from current ones (column 19, lines 7-17). 

Schilit in view of Mishra discloses a user interface for displaying electronic 
information, but does not specifically teach wherein the audio annotations are in the 
form of audio clips, verbally delimiting keywords and dynamically accessing the audio 
clips based on search queries. 

Oliver teaches said audio annotation (voice annotation) in the form of audio clips 
(voice clip) and said audio clips are stored in a storage which includes properties that 
permit audio information to be associated with a visual so that one audio clip is 
displayed with another (image file played back with voice clip; abstract with column 2, 
lines 21-44 and column 3, line 39 - column 4, line 67), to annotate documents with a 
description of the document. 
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Therefore, it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to modify Shilit in view of Mishra's interface wherein the 
audio annotations are in the form of audio clips, as taught by Oliver, to identify the 
particular documents that are desired for the purpose of uploading to a host computer or 
displaying information (column 1, lines 43-54). 

Schilit in view of Mishra and Oliver discloses an interface, but does not 
specifically teach verbally delimiting keywords and dynamically accessing the audio 
clips based on search queries. 

Lucas teaches an interface comprising an audio input (voice controlled; column 
3, lines 38-44) for receiving verbally delimited keywords (delimit) and converting said 
verbally delimited keywords into search queries (abstract with column 7, lines 25-30 and 
column 21, lines 35-67) and wherein the user interface is controlled by a processor 
which allows dynamically accessing the audio clips based on the search queries 
(abstract with column 7, lines 25-30 and column 21 , lines 35-67), to manipulate 
documents. 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to modify Schilit in view of Mishra and Oliver wherein it 
comprises delimiting keywords and dynamically accessing the audio clips based on 
search queries, as taught by Lucas, in order to specify portions of documents to be 
retrieved from a document repository (column 1, lines 36-44). 

Regarding claim 28, Schilit discloses a user interface further comprising: 
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a third display portion for displaying a non-audio annotation (context of each 
annotation; column 6, lines 10-14). 

Regarding claim 31, Schilit discloses a use interface further comprising: 

a third display portion for receiving a user input of a property or properties of said 
audio annotation (attributes; column 5, line 61 - column 6, line 4). 

Regarding claim 32, Schilit discloses a user interface wherein said audio 
annotation is recordable by said user (column 39-42 with column 6, lines 24-26). 

Regarding claim 33, it is interpreted and rejected for the same reasons as set 
forth in claim 27. In addition, Schilit discloses a process for recording an audio 
annotation comprising the steps of: 

displaying electronic information (column 3, lines 52-56); 

receiving a user input (column 6, lines 24-26 with column 3, lines 52-63 and 
column 5, line 45); 

recording an audio annotation in response to said user input (column 4, lines 30- 
38); and 

associating said audio annotation with properties including a displayed portion of 
said electronic information (associates attributes with annotations; column 4, lines 30- 
38), but does not specifically teach wherein said audio annotation is associated with an 
author. 

Mishra teaches a user interface for displaying electronic information to a user 
wherein said audio annotation is associated with an author (column 12, line 59 - column 
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13, line 45 with column 19, lines 8-17 and column 28, line 61 - column 29, line 22), so 

that the marks are distinguishable. 

Therefore, it would have been obvious to one of ordinary skill in the art at the 

time the invention was made to modify Schilit's interface wherein said audio annotation 

is associated with an author, as taught by Mishra, to make it helpful to others to 

distinguish marks made by user A from those of user B and even a user to distinguish 

his or her down earlier remarks from current ones (column 19, lines 7-17). 

Regarding claim 34, Schilit discloses a process further comprising the step of: 
storing said audio annotation prior to the association of said audio annotation 

with said displayed portion (column 5, lines 42-64). 

Regarding claim 35, Schilit discloses a process further comprising the step of: 
storing said audio annotation after the association of said audio annotation with 

said displayed portion (column 5, line 65 - column 6 t line 4). 

Regarding claim 36, Schilit discloses a process wherein said recording step 

records all ambient sounds (audio recording; column 4, lines 39-40 with column 6, lines 

24-26). 

Regarding claim 39, Schilit discloses a process further comprising the step of: 
associating additional properties with said audio annotation at the start of 

recording of said audio annotation (creation date; column 5, lines 61-64). 

Regarding claim 40, Schilit discloses a process wherein one of said properties is 

a file position or document position of an item on said displayed portion of said 

electronic information (column 2, lines 63-66). 
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Regarding claim 41, Schilit discloses a process wherein one of said properties is 

a start identification of said displayed portion of said electronic information (creation 

date; column 5, lines 61-64). 

Regarding claim 42, Schilit discloses a further comprising the steps of: 

storing said audio annotation (stores the annotations; column 3, lines 62-63); and 

searching audio annotations including said audio annotation for at least one 

property matching a query (column 4, lines 30-38). 

Regarding claim 43, it is interpreted and rejected for the same reasons as set 

forth in claim 27. In addition, Schilit discloses a process for playing audio annotations 

comprising the steps of: 

displaying a portion of electronic information (figure 2, element S160 with column 

3, lines 60-63); 

receiving a user input (column 4, lines 39-42 with column 6, lines 24-26); 
assembling said audio annotations into an audio stream (column 4, lines 30-38 
with column 6, lines 24-26); and 

playing said audio stream (column 4, lines 30-38 with column 6, lines 24-26), but does 
not specifically teach wherein said audio annotation is associated with an author. 

Mishra teaches a user interface for displaying electronic information to a user 
wherein said audio annotation is associated with an author (column 12, line 59 - column 
13, line 45 with column 19, lines 8-17 and column 28, line 61 - column 29, line 22), so 
that the marks are distinguishable. 
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Therefore, it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to modify Schilit's interface wherein said audio annotation 
is associated with an author, as taught by Mishra, to make it helpful to others to 
distinguish marks made by user A from those of user B and even a user to distinguish 
his or her down earlier remarks from current ones (column 19, lines 7-17). 

Regarding claim 44, Schilit discloses a process further comprising the step of: 
waiting for a second user input prior to playing said audio stream (column 4, lines 
30-38). 

Regarding claim 45, Schilit discloses a process further comprising the step of: 
playing said audio stream once said audio stream is assembled (column 4, lines 
30-38). 

Regarding claim 46, Schilit discloses a process wherein said user input is a text 
query (text; column 6, lines 24-26). 

Regarding claim 48, Schilit discloses a process further comprising the steps of: 

altering the display of said portion to match a currently playing annotation in said 
audio stream (column 4, lines 30-38 with column 6, lines 24-26). 

Regarding claim 50, it is interpreted and rejected for the same reasons as set 
forth in claim 27. In addition, Schilit discloses a process for playing audio annotations 
comprising the steps of: 

navigating to a page (reader to navigate; column 6, lines 5-9); 

retrieving at least one audio annotation associated with a page or associated with 
an item on a page (column 4, lines 30-38 with column 6, lines 24-32); and 
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playing said at least one audio annotation (column 3, lines 57-63 with column 6, lines 
24-32), but does not specifically teach wherein said audio annotation is associated with 
an author. 

Mishra teaches a user interface for displaying electronic information to a user 
wherein said audio annotation is associated with an author (column 12, line 59 - column 
13, line 45 with column 19, lines 8-17 and column 28, line 61 - column 29, line 22), so 
that the marks are distinguishable. 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to modify Schilit's interface wherein said audio annotation 
is associated with an author, as taught by Mishra, to make it helpful to others to 
distinguish marks made by user A from those of user B and even a user to distinguish 
his or her down earlier remarks from current ones (column 19, lines 7-17). 

Regarding claim 51, Schilit discloses a process further comprising the step of: 

waiting for a user input prior to playing said audio annotation (column 4, lines 30- 
38 with column 6, lines 24-26). 

Regarding claim 52, Schilit discloses a process wherein said item on said page 
includes at least one of embedded notes, inked notes, highlights or underlining (column 
4, lines 44-48). 

Regarding claim 53, Schilit discloses a process wherein said at least one audio 
annotation was previously retrieved and said retrieving step includes indexing said 
previously retrieved at least one audio annotation (column 5, lines 10-32). 
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Regarding claim 54, Schilit discloses a process wherein said at least one audio 
annotation is the result of a newly executed query (column 3, lines 52-63). 

Regarding claim 55, it is interpreted and rejected for the same reasons as set 
forth in claim 27. In addition, Schilit discloses a computer readable medium having a 
data structure stored thereon, said data structure comprising; 

a document (column 3, lines 52-63); 

a link object (column 4, lines 17-38); and 

audio content with at least one property (column 5, lines 61-64), 

wherein said link object references said document and references said audio 
content (column 4, lines 30-38), 

wherein at least one property identifies an author of the at least one audio annotation 
(column 5, line 42 - column 6, line 32), but does not specifically teach wherein said 
audio annotation is associated with an author. 

Mishra teaches a user interface for displaying electronic information to a user 
wherein said audio annotation is associated with an author (column 12, line 59 - column 
13, line 45 with column 19, lines 8-17 and column 28, line 61 - column 29, line 22), so 
that the marks are distinguishable. 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to modify Schilit's interface wherein said audio annotation 
is associated with an author, as taught by Mishra, to make it helpful to others to 
distinguish marks made by user A from those of user B and even a user to distinguish 
his or her down earlier remarks from current ones (column 19, lines 7-17). 
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Regarding claim 56, Schilit discloses a data structure wherein at least one 
property relates to the time said audio content started recording (column 5, lines 55-64). 

Regarding claim 58, Schilit discloses a data structure wherein at least one 
property relates to the length of recording of said audio content (column 4, lines 49-56). 

Regarding claim 60, Schilit discloses a data structure wherein at least one 
property relates to a start ID (column 4, lines 49-56 with column 5, lines 55-64). 

Regarding claim 62, Schilit discloses a data structure wherein said audio content 
is comprised of a plurality of audio clips (column 4, lines 30-38 with column 6, lines 24- 
26). 

Regarding claim 63, Schilit discloses a data structure wherein said audio clips 
are stored in a database (column 3, lines 57-63 with column 6, lines 24-26). 

Regarding claim 66, Schilit discloses a data structure wherein said audio content 
is stored within a document (figure 1, element 16 with column 3, lines 52-63). 

Regarding claim 67, Schilit discloses a data structure wherein said audio content 
is stored apart from a document (figure 1, element 18 with column 3, lines 52-63). 

Regarding claim 68, Schilit discloses a data structure wherein said audio content 
is stored in a database with at least one property designating the position of viewed 
document relating to said audio content (column 4, lines 49-58). 

Regarding claim 69, Schilit discloses a data structure wherein said audio content 
is stored in a database and linked to a separate annotation document that stores the 
position of a viewed document relating to said audio content (column 4, lines 30-38). 
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Regarding claim 70, it is interpreted and rejected for the same reasons as set 
forth in claim 27. In addition, Schilit discloses a process for recording audio content 
comprising the steps of: 

navigating to a page of a document (reader to navigate; column 6, lines 5-9); 

recording said audio content (column 4, lines 39-40); and 

associating properties with said audio content such that retrieval of said audio 
content positions said audio content after previously recorded audio content (column 4, 
lines 30-40), 

wherein said audio content includes at least one audio annotation (column 5, line 
42 - column 6, line 32), but does not specifically teach wherein said audio annotation is 
associated with an author. 

Mishra teaches a user interface for displaying electronic information to a user 
wherein said audio annotation is associated with an author (column 12, line 59 - column 
13, line 45 with column 19, lines 8-17 and column 28, line 61 - column 29, line 22), so 
that the marks are distinguishable. 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to modify Schilit's interface wherein said audio annotation 
is associated with an author, as taught by Mishra, to make it helpful to others to 
distinguish marks made by user A from those of user B and even a user to distinguish 
his or her down earlier remarks from current ones (column 19, lines 7-17). 
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Regarding claim 71, Schilit discloses a process wherein said audio content 
comprises audio clips and properties include a time property (column 5, lines 56-64 with 
column 6, lines 24-26). 

Regarding claim 72, Schilit discloses a process wherein said audio content and 
said previously recorded audio content is ordered at least by said time property 
(creation date; column 5, lines 61-64). 

Regarding claim 73, it is interpreted and rejected for the same reasons as set 
forth in claim 27. In addition, Schilit discloses a process of searching audio clips 
comprising the steps of: 

inputting search terms or properties (column 5, lines 61-64); 

searching said audio clips for said search terms or properties (column 4, lines 30- 
38); and 

ordering audio clips detected by said searching step for output (column 4, lines 30-38 
with column 5, lines 61-64 and column 6, lines 10-23), but does not specifically teach 
wherein said audio annotation is associated with an author. 

Mishra teaches a user interface for displaying electronic information to a user 
wherein said audio annotation is associated with an author (column 12, line 59 - column 
13, line 45 with column 19, lines 8-17 and column 28, line 61 - column 29, line 22), so 
that the marks are distinguishable. 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to modify Schilit's interface wherein said audio annotation 
is associated with an author, as taught by Mishra, to make it helpful to others to 
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distinguish marks made by user A from those of user B and even a user to distinguish 
his or her down earlier remarks from current ones (column 19, lines 7-17). 

Regarding claim 75, it is interpreted and rejected for the same reasons as set 
forth in claim 27. In addition, Schilit discloses a process for recording audio information 
comprising the steps of: 

recording audio signals as a first file (column 4, lines 29-48); 

processing said file to extract audio clips (column 6, lines 24-26); and 

storing said audio clips (column 3, lines 52-63 with column 6, lines 24-26), 
wherein said processing separates the content of said first file into audio clips based on 
events (column 4, lines 30-38), but does not specifically teach wherein said audio 
annotation is associated with an author. 

Mishra teaches a user interface for displaying electronic information to a user 
wherein said audio annotation is associated with an author (column 12, line 59 - column 
13, line 45 with column 19, lines 8-17 and column 28, line 61 - column 29, line 22), so 
that the marks are distinguishable. 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to modify Schilit's interface wherein said audio annotation 
is associated with an author, as taught by Mishra, to make it helpful to others to 
distinguish marks made by user A from those of user B and even a user to distinguish 
his or her down earlier remarks from current ones (column 19, lines 7-17). 
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Regarding claim 77, it is interpreted and rejected for the same reasons as set 
forth in claim 27. In addition, Schilit discloses a process for associating audio notes and 
handwritten notes comprising the steps of: 

creating a handwritten note (inherent in pen-based device; column 3, lines 57-60 
with column 6, lines 24-26); 

associating a time at which said handwritten note was created with said 
handwritten note (creation date; column 5, lines 55-65); 

creating an audio note (column 6, lines 24-26); and 

associating a time at which said audio note was created with said audio note 
(column 5, lines 55-65), 

wherein, upon selection of said handwritten note, audio notes recorded at or near the 
time at which said handwritten note was created are located (column 4, lines 17-56), but 
does not specifically teach wherein said audio annotation is associated with an author. 

Mishra teaches a user interface for displaying electronic information to a user 
wherein said audio annotation is associated with an author (column 12, line 59 - column 
13, line 45 with column 19, lines 8-17 and column 28, line 61 - column 29, line 22), so 
that the marks are distinguishable. 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to modify Schilit's interface wherein said audio annotation 
is associated with an author, as taught by Mishra, to make it helpful to others to 
distinguish marks made by user A from those of user B and even a user to distinguish 
his or her down earlier remarks from current ones (column 19, lines 7-17). 
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Regarding claim 81, Schilit discloses a process wherein said audio notes are 
comprised of audio clips in which each audio clip has a time of creation associated with 
each audio clip (column 5, lines 55-64 with column 6, lines 24-26). 

Regarding claim 82, Schilit discloses a process further comprising the step of: 

playing said audio notes (column 6, lines 24-26). 

Regarding claim 83, it is interpreted and rejected for the same reasons as set 
forth in claim 27. In addition, Schilit discloses a process for playing audio notes 
comprising the steps of: 

displaying a first page of electronic information (presents using different list 
views; column 6, lines 5-9); 

playing audio notes associated with said first page (column 4, lines 30-38); 

displaying a second page of electronic information slide presentation (presents 
using different list views; column 6, lines 5-9); and 

playing audio notes associated with said second page (column 4, lines 30-38), but does 
not specifically teach wherein said audio annotation is associated with an author. 

Mishra teaches a user interface for displaying electronic information to a user 
wherein said audio annotation is associated with an author (column 12, line 59 - column 
1 3, line 45 with column 1 9, lines 8-1 7 and column 28, line 61 - column 29, line 22), so 
that the marks are distinguishable. 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to modify Schilit's interface wherein said audio annotation 
is associated with an author, as taught by Mishra, to make it helpful to others to 
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distinguish marks made by user A from those of user B and even a user to distinguish 
his or her down earlier remarks from current ones (column 19, lines 7-17). 

Regarding claim 84, Schilit discloses a process further comprising the step of 
receiving user input, 

wherein, in response to said user input, said second page is displayed (reader to 
navigate; column 6, lines 5-9). 

Regarding claim 85, it is interpreted and rejected for the same reasons as set 
forth in claim 27. In addition, Schilit discloses a process of recording audio notes 
comprising the steps of: 

displaying a first page of electronic information (column 3, lines 52-63); 

recording a first set of audio notes (column 4, lines 69-40); 

associating said first set of audio notes with said first page (column 4, 1 ines 30- 

38); 

displaying a second page of electronic information (column 3, lines 52-63); 

recording a second set of audio notes (column 4, lines 39-40); and 
associating said second set of audio notes with said second page (column 4, lines 30- 
38), but does not specifically teach wherein said audio annotation is associated with an 
author. 

Mishra teaches a user interface for displaying electronic information to a user 
wherein said audio annotation is associated with an author (column 12, line 59 - column 
13, line 45 with column 19, lines 8-17 and column 28, line 61 - column 29, line 22), so 
that the marks are distinguishable. 
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Therefore, it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to modify Schilit's interface wherein said audio annotation 
is associated with an author, as taught by Mishra, to make it helpful to others to 
distinguish marks made by user A from those of user B and even a user to distinguish 
his or her down earlier remarks from current ones (column 19, lines 7-17). 

Regarding claim 86, Schilit discloses a process further comprising the step of 
receiving user input (column 3, lines 52-63), 

wherein, in response to said user input, said second page is displayed (column 6, 
lines 5-23). 

7. Claims 1, 4-6, 9-10 and 13-26 are rejected under 35 U.S.C. 103(a) as being 
unpatentable by Milne et al. (USPN 5,390,138), hereinafter referenced as Milne in view 
of Mishra in further view of Oliver and Lucas. 

Regarding claim 1, Milne discloses a system for receiving audio input 
comprising: 

a display for displaying electronic information (column 10, line 67 - column 1 1 , 
line 3); 

an audio input receiving audio content (column 10, lines 27-28); and 
a processor (figure 1 , element 10) for associating said received audio content 
with said displayed electronic information (column 8, lines 36-39 with column 10, line 
67 - column 1 1 , line 3), but lacks wherein said processor further associates an author 
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with each of said audio annotations, each of said audio annotations being randomly 
accessible based on the author, wherein the audio annotations are in the form of audio 
clips, verbally delimiting keywords and dynamically accessing the audio clips based on 
search queries. 

Mishra teaches a user interface for displaying electronic information to a user 
wherein said processor further associates an author with each of said audio 
annotations, each of said audio annotations being randomly accessible based on the 
author (column 12, line 59 -column 13, line 45 with column 19, lines 8-17 and column 
28, line 61 - column 29, line 22), so that the marks are distinguishable. 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to modify Milne's interface wherein said audio annotation 
is associated with an author, as taught by Mishra, to make it helpful to others to 
distinguish marks made by user A from those of user B and even a user to distinguish 
his or her down earlier remarks from current ones (column 19, lines 7-17). 

Milne in view of Mishra discloses a user interface for displaying electronic 
information, but does not specifically teach wherein the audio annotations are in the 
form of audio clips, verbally delimiting keywords and dynamically accessing the audio 
clips based on search queries. 

Oliver teaches said audio annotation (voice annotation) in the form of audio clips 
(voice clip) and said audio clips are stored in a storage which includes properties that 
permit audio information to be associated with a visual so that one audio clip is 
displayed with another (image file played back with voice clip; abstract with column 2, 
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lines 21-44 and column 3, line 39 - column 4, line 67), to annotate documents with a 
description of the document. 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to modify Milne in view of Mishra's interface wherein the 
audio annotations are in the form of audio clips, as taught by Oliver, to identify the 
particular documents that are desired for the purpose of uploading to a host computer or 
displaying information (column 1, lines 43-54). 

Milne in view of Mishra and Oliver discloses an interface, but does not 
specifically teach verbally delimiting keywords and dynamically accessing the audio 
clips based on search queries. 

Lucas teaches an interface comprising an audio input (voice controlled; column 
3, lines 38-44) for receiving verbally delimited keywords (delimit) and converting said 
verbally delimited keywords into search queries (abstract with column 7, lines 25-30 and 
column 21, lines 35-67) and wherein the user interface is controlled by a processor 
which allows dynamically accessing the audio clips based on the search queries 
(abstract with column 7, lines 25-30 and column 21, lines 35-67), to manipulate 
documents. 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to modify Milne in view of Mishra and Oliver wherein it 
comprises delimiting keywords and dynamically accessing the audio clips based on 
search queries, as taught by Lucas, in order to specify portions of documents to be 
retrieved from a document repository (column 1, lines 36-44). 
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Regarding claim 4, Milne discloses a system further comprising: 

a storage for storing said audio content with said at least one property (audio can 

be stored; column 9, lines 63-65 with column 19, lines 66-67). 

Regarding claim 5, Milne discloses a system further comprising: 

an input receiving a user's input (column 10, lines 27-28), 

wherein said processor starts recording audio content from said audio input in 

response to said user's input (video tape recorders; column 8, lines 53-60 with column 

10, lines 25-32). 

Regarding claim 6, Milne discloses a system wherein said processor includes a 
voice activated recording system for recording said audio content (column 10, lines 25- 
32 with a record member function; column 19, lines 53-54). 

Regarding claim 9, Milne disclose a system wherein said processor controls said 
display to indicate that audio content is associated with said displayed electronic 
information. 

Regarding claim 10, it is interpreted and rejected for the same reasons as set 
forth in claim 1. In addition, Milne discloses a system for playing audio content, said 
system comprising: 

a display for displaying electronic information (column 10, line 67 - column 11, 
line 3); 

a storage for storing audio content (audio can be stored; column 9, lines 63-65 
with column 19, lines 66-67), said audio content including properties and being 
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associated with said displayed electronic information (column 8, lines 36-39 with 
column 10, line 67 - column 11, line 3); 

an output for outputting at least some of said audio content responsive to 
navigation of said displayed electronic information (column 10, line 66 - column 11, 
line 3); and 

a processor for controlling said display, said storage and said output (figure 1, 
element 10), but lacks wherein said processor further associates an author with each 
of said audio annotations, each of said audio annotations being randomly accessible 
based on the author. 

Mishra teaches a user interface for displaying electronic information to a user 
wherein said processor further associates an author with each of said audio 
annotations, each of said audio annotations being randomly accessible based on the 
author (column 12, line 59 - column 13, line 45 with column 19, lines 8-17 and column 
28, line 61 - column 29, line 22), so that the marks are distinguishable. 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to modify Milne's interface wherein said audio annotation 
is associated with an author, as taught by Mishra, to make it helpful to others to 
distinguish marks made by user A from those of user B and even a user to distinguish 
his or her down earlier remarks from current ones (column 19, lines 7-17). 

Regarding claim 13, Milne discloses a system, wherein said storage is a 
database (audio can be stored; column 9, lines 63-65 with column 19, lines 66-67). 
Regarding claim 14, Milne discloses a system further comprising: 
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an input for receiving a user's input (column 10, lines 27-28), 

wherein said output outputs at least some of said audio content in response to 

receiving said user's input (column 10, line 66 - column 11, line 3). 

Regarding claim 15, Milne discloses a system further comprising: 

an input for receiving a user's input (column 10, lines 27-28), 

wherein said processor searches properties of said stored audio content in 

response to said user's input (video tape recorders; column 8, lines 53-60 with column 

10, lines 25-32). 

Regarding claim 16, Milne discloses a system wherein the output of said 
processor is sent to said display to display an indication of the search results (column 
14, lines 65-68). 

Regarding claim 17, Milne discloses a system wherein the output of said 
controller is sent to the output for playing audio content with properties matching the 
search results (audio data; column 8, lines 47-50 with column 14, lines 65-68 with 
column 21, lines 35-38). 

Regarding claim 18, Milne discloses a system wherein said processor retrieves 
all audio content associated with said electronic information when said electronic 
information is accessed (media components connected together; column 8, lines 36-60 
with column 11, lines 35-36). 

Regarding claim 19, Milne discloses a system wherein said processor outputs 
selected audio content to be played through said output when a page of said electronic 
information is displayed (column 10, line 67 - column 11, line 3). 
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Regarding claim 20, Milne discloses a system wherein said processor 
automatically plays said selected audio content when said page is displayed (audio 
component represented graphically on the display; column 10, lines 38-44 with column 
1 0, line 67 - column 1 1 , line 3). 

Regarding claim 21 , Milne discloses a system further comprising: 

a communication link to transmit said audio content (connection linking audio 
component; column 10, lines 55-56 with column 11, lines 35-36). 

Regarding claim 22, Milne discloses a system further comprising: a network 
connected to said communication link for receiving said audio content, said network 
being accessible by other users (multiple clients to share; column 9, lines 1-4). 

Regarding claim 23, Milne discloses a system further comprising: 

a receiving device of another user for receiving said audio content, said receiving 
device receiving said audio content through one of a wired (connecting devices with 
connecting workstation to network; column 4, lines 24-44) or wireless interface. 

Regarding claim 24, Milne discloses a system wherein said network further 
processes said audio content (processing audio components; column 8, lines 36-50). 

Regarding claim 25, Milne discloses a system wherein said network includes a 
database for storing said audio content (audio can be stored; column 9, lines 63-65 
with column 19, lines 66-67). 

Regarding claim 26, Milne discloses a system wherein said network receives 
audio content without receiving said electronic information associated with said audio 
content (audio and video; column 8, lines 36-60). 
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8. Claim 7 is rejected under 35 U.S.C. 103(a) as being unpatentable over Milne in 
view of Mishra and in further view of Oliver and Lucas, as applied to claim 6, in view of 
Hou et al. (U.S. Patent No. 5,838,313), hereinafter referenced as Hou. 

Regarding claim 7, Milne in view of Mishra and in further view of Oliver and 
Lucas discloses a system wherein said voice activated recording system, but lacks 
wherein the system records when said audio content exceeds a predetermined 
threshold. 

Hou discloses a system wherein the system records when said audio content 
exceeds (is not less than) a predetermined threshold (column 7, lines 63-65), to 
determine distance between previous and new events. 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to modify Milne in view of Mishra and in further view of 
Oliver and Lucas's invention such that it records when said audio content exceeds a 
predetermined threshold as in Hou, to have a report which consists of the individual's 
visual and audio annotations, which can be synchronized for playing back (column 2, 
lines 30-36). 

9. Claim 8 is rejected under 35 U.S.C. 103(a) as being unpatentable over Milne in 
view of Mishra and in further view of Oliver and Lucas as applied to claim 6, in view of 
Dwyer et al. (U.S. Patent No. 6,571,211), hereinafter referenced as Dwyer. 
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Regarding claim 8, Milne in view of Mishra and in further view of Oliver and 
Lucas discloses a system for receiving audio input, but lacks wherein said voice 
activated recording system records when a known user's voice is detected in said 
audio content. 

Dwyer discloses the system wherein said voice activated recording system 
records when a known user's voice is detected in said audio content (column 7, lines 
46-67), so that the users may more readily locate their own voice data files. 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to modify Milne in view of Mishra and in further view of 
Oliver and Lucas's invention such that the recording step records only a specific user's 
voice, to identify an author of a voice data file, which aids in indexing the voice data 
files, so that the users may more readily locate their own voice data files (column 7, 
lines 46-67). 

10. Claim 29 is rejected under 35 U.S.C. 103(a) as being unpatentable over Schilit in 
view Mishra and in further view of Oliver and Lucas and in further view of Headley et al. 
(U.S. Publication No. 2002/0194260), hereinafter referenced as Headley. 

Regarding claim 29, Schilit in view of Mishra and in further view of Oliver and 
Lucas disclose a user interface for displaying electronic information to a user, but lacks 
a third display portion for displaying an indication that said audio annotation is being 
recorded or played back. 
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Headley discloses a user interface further comprising: 

a third display portion for displaying an indication that said audio annotation is 
being recorded or played back (figure 6,element 606), to display additional information. 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to modify Schilit in view of Mishra and in further view of 
Oliver and Lucas's interface wherein it discloses a third display portion for displaying an 
indication that said audio annotation is being recorded or played back, as taught by 
Headley, to display additional information related to each entry in the multimedia playlist 
(column 5, paragraph 0049). 

11. Claim 30 is rejected under 35 U.S.C. 103(a) as being unpatentable over Schilit in 
view Mishra and in further view of Oliver and Lucas and in further view of Aihara et al. 
(USPN 5,644,674), hereinafter referenced as Aihara. 

Regarding claim 30, Schilit in view of Mishra and in further view of Oliver and 
Lucas disclose a user interface for displaying electronic information, but lacks a third 
display portion for displaying one of a document tape or a master tape. 

Aihara discloses a display portion (figure 4, element 33) for displaying one of a 
document tape or a master tape (figure 4, elements 34a-34d with column 16, lines 24- 
33), to view the modified playback picture. 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to modify Schilit in view of Mishra and in further view of 
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Oliver and Lucas's interface, as taught by Aihara, to allow the user to view the modified 
playback picture to confirm whether or not the picture has been modified in the desired 
manner (column 16, lines 24-33). 

12. Claim 36 is rejected under 35 U.S.C. 103(a) as being unpatentable over Schilit in 
view of Mishra and in further view of Oliver and Lucas and in further view of Fielder 
(USPN 6,205,419). 

Regarding claim 36, Schilit in view of Mishra and in further view of Oliver and 
Lucas discloses a system wherein a voice activated recording system, but does not 
specifically teach recording all ambient sounds. ' 

Fielder teaches a continuous recording process that records ambient sounds 
(column 6, lines 41-46), to improve audio quality. 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to modify Schilit in view of Mishra and in further view of 
Oliver and Lucas's process, wherein it records ambient sounds, as taught by Fielder, to 
detect and cancel ambient noise, which provides signal enhancement (column 6, lines 
41-46). 

13. Claim 37 is rejected under 35 U.S.C. 103(a) as being unpatentable over Schilit in 
view of Mishra and in further view of Oliver and Lucas and in further view of Hou et al. 
(U.S. Patent No. 5,838,313), hereinafter referenced as Hou. 
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Regarding claim 37, Schilit in view of Mishra and in further view of Oliver and 
Lucas disclose a system wherein a voice activated recording system, but lacks wherein 
the system records when said audio content exceeds a predetermined threshold. 

Hou discloses a process wherein said recording step records only sounds above 
a predetermined threshold (is not less than the threshold; column 7, lines 63-65), to 
determine distance between previous and new events. 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to modify Schilit in view of Mishra and in further view of 
Oliver and Lucas's invention such that it records only sounds above a predetermined 
threshold, as in Hou, to have a report which consists of the individual's visual and 
audio annotations, which can be synchronizes for playing back (column 2, lines 30-36). 

14. Claim 38 is rejected under 35 U.S.C. 103(a) as being unpatentable over Schilit in 
view of Mishra and in further view of Oliver and Lucas and Hou, as applied to claim 37, 
further in view of Dwyer. 

Regarding claim 38, Schilit in view of Mishra and in further view of Oliver , Lucas 
and Hou disclose a system for receiving audio input, but lacks wherein said recording 
step records only a specific user's voice. 

Dwyer discloses the system wherein said recording step records only a specific 
user's voice (column 7, lines 46-67), so that the users may more readily locate their 
own voice data files. 
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Therefore, it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to modify Schilit in combination with Mishra and in further 
view of Oliver, Lucas and Hou's invention such that voice activated recording system 
records when a known user's voice is detected in said audio content, to identify an 
author of a voice data file, which aids in indexing the voice data files, so that the users 
may more readily locate their own voice data files (column 7, lines 46-67). 

15. Claims 47 and 76 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Schilit in view of Mishra and in further view of Oliver and Lucas and Milne. 

Regarding claim 47, Schilit in view of Mishra and in further view of Oliver and 
Lucas disclose a process for playing audio annotations, but lacks wherein said user 
input is a voice query. 

Milne discloses an object-oriented audio system wherein said user input is a 
voice query (column 10, lines 25-32), to create a variety of interesting applications. 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to modify Schilit in view of Mishra and in further view of 
Oliver and Lucas's process wherein said user input is voice query, as taught by Milne, 
to present a multimedia presentation with a variety of interesting applications (column 
10, lines 21-32). 

Regarding claim 76, Schilit in view of Mishra and in further view of Oliver and 
Lucas disclose a process wherein said events comprise at least one of short pauses in 
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said speech, a pause of a predetermined length, and a user navigating away from a 
displaced page (column 6, lines 5-9), but lacks wherein said audio signals include 
speech. 

Milne discloses an object-oriented audio system wherein audio signals include 
speech (column 10, lines 25-32), to create a variety of interesting applications. 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to modify Schilit in view of Mishra and in further view of 
Oliver and Lucas's process wherein said audio signals include speech, to present a 
multimedia presentation with a variety of interesting applications (column 10, lines 21- 
32). 



16. Claim 49 is rejected under 35 U.S.C. 103(a) as being unpatentable over Schilit in 
view of Mishra and in further view of Oliver and Lucas and in further view of Pritt (U.S. 
Patent No. 5,689,717). 

Regarding claim 49, Schilit in view of Mishra and in further view of Oliver and 
Lucas disclose a process for recording an audio annotation, but lacks including the 
steps of comparing the length and displaying a portion of electronic information. 

Pritt discloses the process including the steps of: 

comparing the length (determining the position) of said currently playing 
annotation with starting identifications of displayable portions of said electronic 
information (column 4, lines 15-30); and 
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displaying the portion of said electronic information (display annotations) 
supporting the greater length of said currently playing annotation (currently displayed; 
column 4, lines 15-30), for the placement of annotations on a computer display of 
various sizes. 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to modify Schilit in view of in view of Mishra and in further 
view of Oliver and Lucas's invention such that it includes the steps of comparing the 
length and displaying a portion of electronic information as in Pritt, for placement of 
annotations of various sizes without overlapping currently displayed annotations 
(column 1, lines 10-15). 

17. Claims 57 and 61 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Schilit in view of Mishra and in further view of Oliver and Lucas and in further view 
Jain et al. (USPN 6,144,375), hereinafter referenced as Jain. 

Regarding claim 57, Schilit in view of Mishra and in further view of Oliver and 
Lucas disclose a computer readable medium having a data structure, but lacks wherein 
at least one property relating to the time said audio content stopped recording. 

Jain discloses a data structure wherein at least one property relates to the time 
said audio content stopped recording (column 19, line 54 - column 20, line 1 1), to 
create a multi-media database. 
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Therefore, it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to modify Schilit in view of Mishra and in further view of 
Oliver and Lucas's medium, as taught by Jain, such that at least one property relates to 
the time said audio content stopped recording, to thereby create a database that 
synchronizes and associates multiple multi-media data types with multi-media events of 
interest to an end user or client (column 20, lines 4-11). 

Regarding claim 61, Schilit in view of Mishra and in further view of Oliver and 
Lucas disclose a computer readable medium having a data structure, but lacks a data 
structure wherein at least one property relates to a stop ID. 

Jain discloses a data structure wherein at least one property relates to a stop ID 
(column 19, line 54 - column 20, line 1 1), to create a multi-media database. 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to modify Schilit in view of Mishra and in further view of 
Oliver and Lucas's medium wherein at least one property relates to a stop ID, as taught 
by Jain, to thereby create a database that synchronizes and associates multiple multi- 
media data types with multi-media events of interest to an end user or client (column 20, 
lines 4-11). 

18. Claim 59 is rejected under 35 U.S.C. 103(a) as being unpatentable over Schilit in 
view of Mishra and in further view of Oliver and Lucas and in further view Hurtado et al. 
(USPN 6,418,421), hereinafter referenced as Hurtado. 
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Regarding claim 59, Schilit in view of Mishra and in further view of Oliver and 
Lucas disclose a computer readable medium having a data structure, but lacks wherein 
said property relates to the author of the recording. 

Hurtado discloses a data structure wherein said property relates to the author of 
the recording (column 42, lines 40-43), to obtain a unique value that is assigned to the 
entity of creation. 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to modify Schilit in view of Mishra and in further view of 
Oliver and Lucas's medium, as taught by Hurtado, wherein said property relates to the 
author of the recording to validate the authenticity and integrity of the contents (column 
42, lines 5-43). 

19. Claims 64 and 65 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Schilit in view of Mishra and in further view of Oliver and Lucas and in further view 
of Martin et al. (USPN 6,272,484), hereinafter referenced as Martin. 

Regarding claim 64, Schilit in view of Mishra and in further view of Oliver and 
Lucas disclose a data structure of audio content, but lacks wherein said property is one 
of a plurality of properties and wherein at least one properties are in a marked up 
language form. 

Martin discloses an electronic document wherein said property is one of plurality 
of properties and wherein at least on of a plurality properties are in a marked up 
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language form (column 6, lines 19-36), to use various applications on a computer 
system. 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to modify Schilit in view of Mishra and in further view of 
Oliver and Lucas's structure wherein said property is one of plurality of properties and at 
least one of a plurality properties are in a marked up language form, as taught by 
Martin, to have a variety of formats to view (column 6, lines 19-36). 

Regarding claim 65, Schilit in view of Mishra and in further view of Oliver and 
Lucas disclose a data structure of audio content, but lacks wherein said properties are 
in XML 

Martin discloses an electronic document wherein said properties are in XML, 
(column 6, lines 19-36), to use various applications on a computer system. 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to modify Schilit in view of Mishra and in further view of 
Oliver and Lucas's structure wherein said properties are in XML, as taught by Martin, to 
have a variety of formats to view (column 6, lines 19-36). 

20. Claim 79-80 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Schilit in view of Mishra and in further view of Oliver and Lucas and in further view of 
Miller et al. (USPN 5,801,685), hereinafter referenced as Miller 
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Regarding claim 79, Schilit in view of Mishra and in further view of Oliver and 
Lucas disclose a process for associating audio notes and handwritten notes, but lacks 
locating said audio notes includes the step of searching a table. 

Miller discloses a process wherein locating said audio notes includes the step of 
searching a table (column 15, lines 29-43 and column 16, lines 11-14), to indicate the 
location of the clip. 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to modify Schilit in view of Mishra and in further view of 
Oliver and Lucas's process, such as taught by Miller, wherein locating said audio notes 
includes the step of searching a table, to indicate the location of the clip (column 15, 
lines 29-43), for easy access, which is well know in the art. 

Regarding claim 80, Schilit in view of Mishra and in further view of Oliver and 
Lucas disclose a process for associating audio notes and handwritten notes, but lacks 
locating said audio notes includes the step of searching a linked list. 

Milller discloses a process wherein locating said audio notes includes the step of 
searching a linked list (column 9, lines 34-55 with column 1 1 , lines 14-42), to obtain the 
raw information that resides in the server. 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to modify Schilit in view of Mishra and in further view of 
Oliver and Lucas's process, such as taught by Miller, wherein locating said audio notes 
includes the step of searching a linked list, to allow information to be obtained (column 
1 1 , lines 14-42), with easy access, which is well known in the art. 
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Conclusion 

21 . The prior art made of record and not relied upon is considered pertinent to 
applicant's disclosure. 

Golovchinsky et al. (PGPUB 2002/0083094) disclose a method and apparatus for 
annotating widgets. 

22. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Jakieda R. Jackson whose telephone number is 571- 
272-7619. The examiner can normally be reached on Monday-Friday from 5:30am- 
2:00pm. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, David Hudspeth can be reached on 571-272-7843. The fax phone number 
for the organization where this application or proceeding is assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from the 

Patent Application Information Retrieval (PAIR) system. Status information for 

published applications may be obtained from either Private PAIR or Public PAIR. 

Status information for unpublished applications is available through Private PAIR only. 

For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 

you have questions on access to the Private PAIR system, contact the Electronic 

Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a 

USPTO Customer Service Representative or access to the automated information 

system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 
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